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Executive  Summary 


This  report  defines  the  minimum  required  conditions  necessary  for  qualifying  payload  software  prior 
to  launch  or  use  on  orbit. 

•  The  recommended  approach  is  to  complete  full  payload  system  (software,  hardware,  inte¬ 
gration  and  external  interface)  qualification  prior  to  launch. 

•  If  launch  will  take  place  before  full  payload  qualification,  then 

-  Core  payload  software  functionality  (i.e.,  bootstrap  startup,  spacecraft  communi¬ 
cations,  system  upload,  hardware/software  status  reporting,  and  safing)  must  be 
fully  qualified  before  launch. 

-  Incremental  upload  of  the  software  must  be  planned,  architected,  designed,  and 
exhaustively  tested. 

-  Core  software  must  be  integrated  and  tested  with  all  unqualified  software  using  a 
qualified  test  environment  prior  to  upload. 

In  either  case,  full  qualification  of  both  core  and  basic  (i.e.,  non-core)  payload  software  must  be  com¬ 
plete  before  on-orbit  use. 

Independent  verification  &  validation  is  a  method  to  reduce  risk  and  increase  the  reliability  of  the 
payload  software. 


iii 


Acknowledgment 


The  authors  would  like  to  thank  the  following  members  of  the  Computer  and  Software 
Division  for  their  comments  and  assistance  in  preparing  this  report. 

Richard  Adams 
Douglas  Buettner 
Lance  Diemback 
Suellen  Eslinger 
Leslie  Holloway 
Larry  Jansen 
Lee  Marvin 
Mary  A.  Rich 
Marilee  Wheaton 


v 


Contents 


1 .  Scope .  1 

2.  Definition  of  Terms .  3 

2.1  Software  Qualification  Testing .  3 

2.2  Payload  Software .  3 

3.  Payload  Software  Functions .  5 

4.  Testing  the  Payload  Software .  9 

4.1  Software  Qualification  Testing  Scope .  9 

4.2  Testing  Environment . 10 

5.  Conclusion .  13 

References .  15 

Figure 

1  Payload  qualification  process  flow .  7 

Table 

1 .  Payload  Capabilities .  5 


vii 


1.  Scope 


This  report  defines  the  minimum  required  conditions  necessary  for  qualifying  payload  software  prior 
to  launch  or  use  on  orbit.  This  minimum  set  of  conditions  must  be  satisfied  for  Aerospace  to 
recommend  launch  of  or  use  of  payload  software  on  orbit. 
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2.  Definition  of  Terms 


2.1  Software  Qualification  Testing 

Test  Requirements  For  Launch,  Upper-Stage,  and  Space  Vehicles1  states  “Software  qualification 
testing  verifies  that  the  software  meets  its  specified  requirements.”  Reference  1  goes  on  to  provide 
several  important  comments  on  software  qualification  testing. 

•  “Software  qualification  testing  completion  criteria  shall  include  the  successful  execution 
of  software  qualification  test  cases  covering,  as  a  minimum,  verification  of  all  software 
requirements  under  conditions  that  are  as  close  as  possible  to  those  that  the  software  will 
encounter  in  the  operational  environment  (e.g.,  operational  flight  data  constants,  opera¬ 
tional  input  and  output  data  rates,  target  flight  hardware  configurations); 

•  verification  of  all  software  interface  requirements,  using  the  actual  interfaces  wherever 
possible  or  high-fidelity  simulation  of  the  interfaces  where  not  possible; 

•  verification  of  all  software  specialty  engineering  requirements  (i.e.,  supportability,  test¬ 
ability,  reliability/maintainability /availability,  safety,  security,  as  applicable),  including  in 
particular  verification  of  software  reliability  requirements  and  fault  detection,  isolation, 
and  recovery  requirements; 

•  stress  testing,  including  worst-case  scenario(s);  and  resource  utilization  measurement 
(e.g.,  CPU,  memory,  storage,  bandwidth).” 

The  testing  and  verification  above  falls  into  the  category  of  ”requirements-based”  testing.  These  tests 
include  full  path  coverage  at  the  unit  level,  interface  testing  in  accordance  with  the  applicable  Inter¬ 
face  Control  Documents  (ICDs),  nominal  and  off-nominal  testing,  and  stress  testing.  Full  qualifica¬ 
tion,  as  used  in  this  report,  includes  all  the  above  testing  and  verification  efforts  performed  on  the 
payload  hardware  and  software  together  with  robustness  testing,3  which  characterizes  how  the  pay- 
load  software  responds  to  invalid  input  parameters.* 


2.2  Payload  Software 

Payload  software  is  a  part  of  flight  software,  that  is,  all  the  software  on  the  satellite  while  on  orbit. 
Payload  software,  as  the  term  is  used  in  this  report,  refers  to  all  software  used  on  orbit  by  the  payload 
hardware.  Such  software  includes  any  firmware  used  to  directly  control  hardware  as  well  as  the  pro¬ 
grams  used  in  any  form  on  any  processors  included  with  the  payload.  Payload  software  also  includes 
all  tables  and  parameters  kept  onboard  and  accessed  by  the  payload  software  to  perform  its  functions. 


This  report  does  not  deal  with  other  methods  of  software  qualification,  such  as  inspection  or  analysis.  While  those 
methods  are  important,  they  are  not  covered  in  this  report. 
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For  the  purposes  of  this  report,  payload  software  does  not  include  any  functions  performed  by  the 
spacecraft  for  its  own  purposes  or  on  behalf  of  the  payload. 
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3.  Payload  Software  Functions 


Table  1  lists  the  core  and  basic  functions  performed  by  payload  software.  Because  payload  capabili¬ 
ties  vary  tremendously  based  on  their  mission,  this  set  of  capabilities  must  be  evaluated  for  each  sys¬ 
tem  to  ensure  completeness.  The  terminology  used  will  vary  from  mission  to  mission  and  contractor 
to  contractor,  but  these  capabilities  will  be  present  in  all  payloads.  Table  1  does  not  give  a  special 
entry  to  any  software  or  firmware  that  cannot  be  uploaded  after  launch.  Such  software  or  firmware 
must  be  considered  a  possible  single-point  failure,  and  should  be  treated  accordingly. 

The  first  five  capabilities  in  Table  1  comprise  the  core  set  of  payload  capabilities  and  are  critical  for 
minimum  payload  operation.  If  any  of  these  five  capabilities  fail,  the  result  could  range  from 
degraded  payload  performance  to  total  payload  failure.  If  the  five  core  capabilities  have  been  fully 
tested  and  qualified  as  defined  in  Subsection  4.1  of  this  report,  it  is  possible  to  correct  problems  in  the 
remaining  functions  by  uploading  modified  software.  To  qualify  any  software,  a  consistent  logical 
progression  of  testing,  in  accordance  with  good  software  engineering  practice,  starting  with  unit  test¬ 
ing  to  integration  testing  to  system  testing,  must  be  completed.  These  tests  must  be  thorough  and 
include:  path  and  thread  testing;  off-nominal  cases;  boundary  conditions;  stress  cases;  and  robustness 
testing3  Robustness  testing  characterizes  how  the  payload  software  responds  to  invalid  input 
parameters. 

The  last  two  basic  capabilities  are  important  for  the  payload’s  mission.  However,  the  core  set  com¬ 
prises  those  functions  requiring  qualification  to  allow  modification  to  the  core  and  basic  capabilities, 
whether  this  modification  is  due  to  errors  in  the  software  or  the  desire  to  have  the  payload  perform 
functions  that  were  not  included  in  the  original  design  or  previous  uploads. 

The  Bootstrap  Startup  capability  is  needed  to  enable  ground  controllers  to  restart  the  processor  and 
bring  the  payload  into  a  known,  safe  state.  The  Bootstrap  Startup  can  be  initiated  by  the  spacecraft, 
the  ground  controllers,  or  a  cold  boot  startup  initiated  by  the  payload  (including  the  operating  system 
and  a  restart  command  on  the  payload  that  forces  a  reboot).  It  may  also  perform  some  initial  hard¬ 
ware  checks  (e.g.,  memory  testing)  and  reporting  of  the  results  to  the  spacecraft  for  telemetry  down¬ 
link.  From  this  known,  safe  state,  ground  controllers  can  direct  the  next  step  for  the  payload.  This 
could  be  an  upload  of  the  operational  payload  software,  a  jump  to  start  payload  operation  if  the  sofit- 


Table  1 .  Payload  Capabilities 


1 

Bootstrap  Startup 

Core 

2 

Spacecraft  Communications 

Core 

3 

System  Upload 

Core 

4 

Hardware/Software  Status  Reporting 

Core 

5 

Sating  the  payload 

Core 

6 

Hardware  Control 

Basic 

7 

Payload  Data  Processing 

Basic 
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ware  is  already  onboard,  or  starting  some  diagnostic  mode  if  necessary.  If  the  Bootstrap  Startup  does 
not  operate  properly,  it  will  be  impossible  to  put  the  payload  into  a  known,  safe  state  and  may  lead  to 
payload  mission  failure  since  the  ground  controllers  may  be  unable  to  start  the  operational  payload 
software. 

The  System  Upload  capability  is  necessary  to  reprogram  the  payload  processor  and  other  reprogram¬ 
mable  devices  in  the  payload,  as  well  as  provide  updated  payload  data.  This  function  is  essential  for 
remedying  onboard  software  faults  or  for  providing  payload  software  changes  required  to  the  existing 
onboard  software  over  time.  This  capability  decreases  the  probability  that  the  payload  software  can 
place  the  payload  in  an  unrecoverable  state. 

The  Spacecraft  Communications  capability  includes  any  communications  involving  commanding  of 
the  payload  and  receiving  the  payload  response.  This  report  assumes  that  the  spacecraft  provides  all 
communications  services  between  the  ground  and  the  payload.  If  the  payload  cannot  communicate 
with  the  ground  controllers,  it  will  be  unable  to  provide  data  for  downlink  to  the  ground  and  incapable 
of  receiving  ground  commands  such  as  uploading  the  software.  If  this  communications  path  does  not 
operate  correctly,  the  payload  is  useless. 

The  Hardware/Software  Status  Reporting  capability  provides  health,  status,  and  debug  information  to 
the  ground.  This  information  is  more  comprehensive  than  the  information  provided  during  Bootstrap 
Startup  and  is  intended  to  provide  the  ground  with  sufficient  information  to  troubleshoot  problems, 
allow  trending  of  parameters  and  the  like. 

The  Safing  function  ensures  that  the  payload  can  be  brought  into  a  known,  safe  state  that  will  pre¬ 
serve  the  payload  from  the  environment  in  the  event  of  problems.  This  can  be  activated  either  by  a 
ground  command  or  due  to  a  problem  in  the  payload  itself.  It  is  included  as  part  of  the  core  set  to 
ensure  that  the  payload  can  be  brought  into  a  known,  safe  state  from  any  other  payload  state.  Once 
the  payload  has  entered  a  Safe  Hold  state,  it  is  critical  that  the  payload  can  be  commanded  to  transi¬ 
tion  out  of  Safe  Hold  into  a  test,  operational,  or  shutdown  state. 

In  summary,  these  five  core  capabilities  must  be  fully  qualified  prior  to  launch.  Without  Spacecraft 
Communications  or  the  Bootstrap  Startup,  the  payload  will,  not  function,  and  without  System  Upload, 
the  payload  may  not  function,  as  it  would  be  impossible  to  correct  any  software  errors  found  in  the 
payload  software.  If  the  Hardware/Software  Status  Reporting  fails,  the  ground  may  not  be  able  to 
determine  whether  any  of  the  other  functions  are  operating  as  intended  and  may  not  be  able  to  verify 
that  data  coming  from  the  payload  is  correct.  After  executing  the  Bootstrap  Startup  function,  the 
payload  is  in  a  known,  safe  state.  If,  after  transitioning  from  this  state  into  a  mission  state,  the  pay- 
load  hardware  or  software  encounters  a  problem  that  it  is  not  prepared  for,  there  must  be  a  method  to 
return  to  a  known,  safe  state.  That  is  the  responsibility  of  the  Safing  function.  If  this  function  fails  to 
operate  properly,  the  payload  may  be  exposed  to  conditions  that  degrade  its  ability  to  perform  the 
mission.  The  Safing  function  can  be  triggered  by  payload  hardware  or  software,  the  spacecraft,  or  the 
ground  controllers. 

While  it  is  not  a  foregone  conclusion  that  any  of  the  core  capabilities  will  fail  without  being  fully 
qualified,  the  risk  involved  is  viewed  as  unacceptably  high.  Qualification  testing  of  these  core  func- 


6 


tions  ensures  that  the  best  effort  has  been  made  to  eliminate  problems  that  would  jeopardize  mission 
success  while  it  is  still  possible  to  remedy  any  such  problems.  After  launch,  correcting  some  errors 
may  prove  impossible. 


The  remaining  two  basic  functions,  Hardware  Control  and  Payload  Data  Processing,  comprise  the 
remainder  of  the  payload  functions.  The  Hardware  Control  capability  ensures  that  the  various  hard¬ 
ware  subsystems  of  the  payload  are  functioning  as  required.  Payload  Data  Processing  ensures  that  the 
data  provided  by  the  hardware  are  processed  and  prepared  for  downlink  via  the  satellite  bus.  As 
stated  previously,  if  the  five  core  capabilities  have  been  fully  qualified,  it  is  possible  to  correct  prob¬ 
lems  in  the  remaining  functions  by  uploading  modified  software.  It  may  also  be  necessary  to  modify 
the  software  for  the  core  functions.  Such  modifications  to  the  core  set  would  require  a  hardware 
design  that  allows  for  the  modification  of  the  core  software,  e.g.,  storing  the  core  set  of  functions  in 
field  programmable  circuitry,  and  require  that  the  core  functions  be  operating  properly,  or  in  some 
degraded  but  functional  mode  that  would  still  permit  uploading  of  software. 


As  shown  in  Figure  1,  the  qualification  of  flight  hardware  and  simulator  software  is  expected  to  occur 
in  parallel.  The  initial  software  qualification  is  targeted  to  achieving  full  qualification  of  the  core  set 
prior  to  launch.  Once  on  orbit,  only  the  qualified  payload  software  can  be  activated.  Operational 
data  will  be  sent  to  the  ground,  generated  by  the  qualified  software,  for  analysis,  and,  if  necessary, 
improvements  made  to  the  software  and  payload  simulator.  As  new  or  modified  software  is  intro¬ 
duced,  the  new  system  must  be  fully  regression  tested  prior  to  upload.  This  process  of  software  quali¬ 
fication  must  be  iterated  until  all  functionality  has  been  developed  and  fully  qualified  before  upload. 


Figure  1  Payload  qualification  process  flow. 
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4.  Testing  the  Payload  Software 


4.1  Software  Qualification  Testing  Scope 

The  following  types  of  testing  lead  to  full  qualification  of  the  software: 

•  Interface  (according  to  the  ICDs) 

•  Nominal  (including  boundary  testing) 

•  Off-nominal  (out  of  bounds  testing) 

•  Stress 

•  Robustness 

•  Full  regression  (for  any  modifications) 

The  various  testing  types  should  be  applied  to  the  software  at  the  appropriate  phase  of  development. 
That  is,  while  off-nominal  testing  would  be  applied  at  the  unit-testing  phase,  it  is  just  as  suitable  at  the 
system  level,  only  with  different  inputs.  Some  of  these  tests  may  not  be  possible  in  certain  phases; 
for  example,  stress  testing  does  not  make  much  sense  at  the  unit  level.  Robustness  testing  is  included 
to  cover  those  testing  cases  to  ensure  that  only  qualified  software  can  be  activated  under  any  circum¬ 
stance  and  that  no  unqualified  software  can  be  activated.  This  verifies  that  the  core  set  (plus  any 
other  qualified  software)  is  adequately  decoupled  from  the  remaining,  unqualified  software. 

As  new  functionality  is  added  to  the  existing  software  base,  qualification  testing  of  the  new  function¬ 
ality  must  be  performed  in  concert  with  requalification  of  the  existing  functionality.  That  is,  adding 
or  changing  software  requires  not  only  the  full  gamut  of  testing  cited  above  for  the  new,  added  soft¬ 
ware  but  also  that  regression  testing  be  performed  on  both  the  new  or  modified  software  and  the 
existing  capabilities  to  ensure  that  the  new  functions  do  not  have  a  negative  effect  on  the  existing, 
qualified  functions.  This  regression  testing  must  include  all  the  qualification  testing  previously  per¬ 
formed  on  the  core  set,  and  a  representative  subset  of  the  testing  performed  on  any  basic  capabilities 
previously  qualified. 

Proper  documentation  of  the  successful  completion  of  the  testing  performed  during  the  appropriate 
phase  of  software  development  is  an  essential  element  of  full  qualification  testing.  Such  artifacts 
demonstrate  the  development  organization’s  commitment  to  logical,  consistent,  suitable  software 
processes;  provide  the  necessary  information  for  maintenance  of  the  software;  and  increase  confi¬ 
dence  that  the  payload  software  will  perform  as  desired  on  orbit. 
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4.2  Testing  Environment 

Software  qualification  testing  ideally  is  performed  using  the  actual  flight  target  software  and  hard¬ 
ware  (hardware  in  the  loop)  in  a  configuration  as  close  as  possible  to  the  operational  configuration. 
While  spacecraft  bus,  gyro,  and  other  simulators  may  be  used,  all  capabilities  must  be  tested  on  real 
hardware  (flight  computers  in  flight  configuration),  either  with  the  actual  flight  hardware  or  engi¬ 
neering  flight-like  models  with  well-known  and  understood  deltas  to  the  actual  flight  software  and 
hardware.  These  capabilities  must  also  have  undergone,  and  passed,  all  system-level  testing.  This 
system-level  testing  must  include  an  end-to-end  test,  including  the  ground  software  and  hardware  to 
ensure  that  the  payload  cannot  just  communicate  with  the  spacecraft  but  that  this  communication 
extends  to  the  ground  as  well. 

Software  qualification  testing  specifies  the  use  of  actual  interfaces  wherever  possible  or  high-fidelity 
simulation  of  the  interfaces  where  not  possible.  If  hardware  is  not  available,  simulators  must  be  used. 
These  simulators,  together  with  any  hardware  in  the  loop,  must  be  qualified  if  they  are  to  be  used  to 
qualify  payload  software.  The  software  and  hardware  qualification  testing  for  any  simulator  is  to 
ensure  that  the  test  system  is  flight-like  with  known  and  well-understood  deltas  to  the  actual  flight 
software  and  hardware. 

An  engineering  model,  as  defined  in  this  report,  comprises  as  much  flight  hardware  as  feasible  and  is 
intended  to  emulate  the  actual  payload  and  spacecraft  interfaces.  The  engineering  model  must  also 
include  simulators  for  any  subsystems  that  are  not  available.  A  simulator  is  an  implementation  of  a 
subsystem  that  may  later  be  built  in  hardware  or  software. 

In  testing,  actual  flight  hardware  is  preferable  to  engineering  models,  and  engineering  models  are 
preferable  to  high-fidelity  simulators.  While  low-fidelity  simulators  may  be  used  early  in  the  soft¬ 
ware  development  and  integration  phases,  such  simulators  are  not  considered  adequate  once  the  pro¬ 
ject  reaches  the  software  qualification  phase.  If  flight  hardware  used  in  ground  testing  does  not  pro¬ 
vide  data  adequate  for  qualification  testing  of  the  payload  software,  either  engineering  models  or 
high-fidelity  simulators  must  be  used. 

A  testbed  is  a  combination  of  hardware  and  software  used  to  simulate  the  payload  and  its  on-orbit 
environment.  It  includes  a  flight-like  payload,  the  engineering  model  or  a  high-fidelity  payload  simu¬ 
lator,  together  with  the  interfaces  that  the  payload  will  use  on  orbit.  These  interfaces  may  be  either 
hardware  or  software  or  a  combination  of  both.  It  is  imperative  that  the  contractor  provide  a  testbed 
of  some  type,  separate  from  the  testbed  used  for  software  development.  A  good  testbed  is  necessary 
during  the  integration  and  testing  phase  as  it  allows  testing  and  software  development  to  proceed  in 
parallel  and  it  permits  testing  of  functionality  in  ways  not  convenient  on  the  flight  hardware.  After 
launch,  the  testbed  provides  a  facility  for  continuing  testing,  both  of  newly  developed  software  and 
for  qualification  purposes,  and  anomaly  resolution.  Before  a  problem  can  be  solved,  it  must  be  diag¬ 
nosed,  and  the  root  cause  well  understood.  The  testbed  also  allows  changes  to  the  payload  software 
to  be  tested  thoroughly  before  being  uploaded  to  the  actual  payload. 

An  ideal  testbed  would  contain  a  flight-worthy  payload  hardware  system  and  the  supporting  software. 
Using  a  real  payload  in  this  role  is  rarely  possible.  The  use  of  engineering  models  is  the  most  likely 
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second  choice.  Problems  may  arise  if  the  engineering  model  does  not  use  flight-like  hardware,  for 
example,  a  slower  processor  than  the  one  used  on  the  flight  unit.  Such  a  processor  will  skew  per¬ 
formance  testing  of  the  payload.  The  testbed  will  make  use  of  the  various  simulations  mentioned 
above.  A  strong  lesson  learned  is  that  the  overall  quality  and  fidelity  of  the  test  environment  have  a 
direct  impact  on  software  quality,  reliability,  and  risk. 
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5.  Conclusion 


This  report  has  defined  the  minimum  required  conditions  necessary  for  qualifying  payload  software 
prior  to  launch  or  use  on  orbit.  The  ideal  qualification  process  involves  qualifying  all  payload  soft¬ 
ware  before  launch  following  the  standard  practices  described  in  References  1 ,  2,  and  3 .  If  launch 
takes  place  before  all  payload  software  is  qualified,  then  the  alternate  path  within  the  qualification 
process  involves  qualifying  as  much  software  as  possible  to  include  all  of  the  core  payload  software 
functionality  (i.e.,  bootstrap  startup,  spacecraft  communications,  system  upload,  hardware/software 
status  reporting,  and  safing)  before  launch,  activating  this  core  when  the  payload  is  on  orbit,  qualify¬ 
ing  non-core  payload  software  on  the  ground  using  a  qualified  payload  simulator,  and  uploading  and 
activating  the  non-core  payload  software.  Regardless  of  which  path  is  taken,  this  qualification  proc¬ 
ess  requires  that  the  core  and  non-core  payload  software  be  qualified  before  being  used  on  orbit.  It 
should  also  be  stated  that  even  fully  qualified  software  could  experience  anomalies  on  orbit.  The  core 
capabilities  must  allow  for  effective  troubleshooting  from  the  ground.  ^ 


All  core  payload  software  must  be  fully  qualified  before  being  launched,  and 
all  payload  software  must  be  qualified  before  being  used  on  orbit. 
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